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health professional education programs (D/E HPEPsV as a means of 
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intended to serve as background material fdr'the development of a 
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drawn from conference proceedings and prepared papers of Health 
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.' ■ EVALUATION OF DECENTf^I^^ AND REc^fOi^AMS ^^t^ . 
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'INTRODLiQ^toN 



The- Federal Government haS $^Pported he^j_tb ^Ower 

cation and training for many yaaf^- Initial;j^y', > '^^ai supP^^^ 

of medical schools primarily tooK tj^^ form 0;c gf for bi^"' 

° / ' abi 

'medical research. The large xas^^^ch P^og^^tti^ " medi^^'^ 

schools to attract well-qualif ie<J ^acul^i^s ^ ^^^t^t te^<^^^^^ 

programs. The, ^"Health PTof es^ioii^" ^^jucation^]. A '. "^anee A^^^ 

of 1963 (P.L. 88 -129)- pro^vided f^^ niore dire^^^ S^^^^^t of 

, ' , ^ . / ^ f O V - J Tl"- 

teachinl functions by^ authoriziag "hatching g:t^^n1? ^ the '^cp^^ 
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struction of teaching facilities loans fj^j- 5 'it.^ "cif 

* ^ ■ ' '^^i 

meditine, osteopathic medicin^;»» 

^T^a dentigtr^^ leg la^^ 

was primarily concerned wi th. incf ^^s ing'. ^he i^i;inb s 5 grad^^ 
from the nati^n^'s heal th ipro fe^s i^^s schools^ p^^^H^ 
uneven geographic distribution^ rapi*^ de^i^in^ ^ th^^ n^^^^^. 

of primary care .practitioners , a]^<^ the incre^^m^ ^^^ber o£ 
foreign medical graduates fillia^ imary * ca^^^ t ^Xicy ' 
positions were recggnized as imp^^^ant i^sue^ ^ ' ; ... 

After a. number of years o£ ^^^s ""type of ^up^ * it ha^ 
been recognized that simply ifiCve^^ij^g the n^^be^ Physici^^^ 
and -er health personnel is- not effectiv-g nJ^^^^ of s'ol^^^^'' 
health manpower distribution prroV^^Mi '. In r^^^en "^^s , .th^ 

: ■ ' . - ■ ■ 4 

i ■ \ • ^ . 



nature of ' Federal involvement in healt^ manpower education has 
changed in order to address, problems of geographic' and specialty 
distribution more directly. Federal funds are now supporting 
more specialized programs such as "contingency" loans for^stu- 
dents who agree to practice for a number of y,ears in health 
manpower shortage areas, family practice residency programs, and 
various decentralized and regionalized health professional- edu- 
cation programs, / ■ ' /' 

^ The .rationale for this type of Federal intervention tests 
oh several -^rceptions which are yidely regarded as axion/atfic. 
The first of these is that the current geographic and specialty 
, dis'tribution of health workers i^ unsatisfactory; that the 
quality of the nation's health |are suffers a? a'conseouence '^f 
the current distributional pattern. To improve the si/tu^tion, 
more physicians and allied health personnel are needed in urban 
and rural' medically underserved are.as, and relatively^ more 
emphasis on primary medical care is needed. .# second -perception 
is that these distributional problems do not seem td be s^f- 
correcting. The existing incentive structure sui^ro4nd.ing choice 
of practice location and specialty or type of practice serves 
^to perpetuate and even to Aggravate tiie problems 

• 'Third, it is believed that focused outside efforts can 

/ " 

affect health manpower distribution, and that the^ problems are 
sufficiently serious to warrant attention from' tlie Federal 
Government. And f inal ly , ' rel ated to the new typ^es ^o f , programs 
now receiving Federal support, it is thought that interventions 



at the ejducationa*! level are likely to be effective in influencing 
lacation and special ty decisions. 

This paper focuses on one of thes.e specialized types of 
intervention: decentralized and regionalized health ^professional 
education programs. (D/.R HPEPs}. The discussion is intended- to 
serve as background material for the developmen-t of a plan to 
evaluate D/R HPEPs. The ideas and observations presented here 
are drawn from conference proceedings and prepared papers of 
Health -Resources Admin^rSTration staff a d members of «the Advisory 
Group on Evaluation of D/R HPEPs.* ' 

Th^ first part of the paper deals with descriptive-- aspects 
.of D/R HPEPs. Topics which ^e considered include: » - " ^ 

• The D/R HPEP Concept ; • ^ 

• Types of D/R HPEPs ^ ^ ^ 

• Goals and Objectives of D/R HPEPs 
; ^ D/R HPEP Activities 

The latter. part of the paper de.als with evaluation- concerns ^ 
including reasons .-for evaluating D/R HPEPs and issues which musA 
be considered in designing an evaluation. . 



These papers wer'b prepared for a works'hop on Evaluation of 
Decentrali'Zed Health* Professional Education held Aug. 24-26,. 1975 



CRIPTION OF DECEiVTRALI ZED/REGIONALIZED 
ALTH PROFESSIONAL. EguCATION. PROGRAMS ; 

^ Jhe Concept' of Dedentrali zed/ Regional^ey Health Trofessipnal 
Education 




• The D/R BPEP concept can be 4^scribed in terms of three' 
•central ideas. Decentralization refers to a •relocation of hea^lth 

. - - ■ ' — ^ • ■ :^ 

manpowef^ education activities, especially physician training' 
progranins, ^to medically uhderserved areas, Decentralized pro- 
grams are intended to reproduce, in all esentfal character is tig^^ , 
programs which are conducte'd^ at the' medical' schools or UniversvCy^ 
Health Science^ Center , It is hoped that the relocation of 
training" ^^grdms will induce students to remai]^ in 'small com- 
munities 'to practice. It is also /felt that clinical training ' 
at sites remote from .the University Health Science Center i^^SC) ^/ 
may mbre successfully be able to promote a primary care orientation:. 
Regional i z at ion refers, to the coordination of exis|j0ig/-"" ^ ^ 
local educational pyog.rams with other €?ducational programs and 
with local delivery institutions , and the creation of new* pro - 
g.rams to meet local n^eds. In general, thoughvthere are 

4 - ' 

i exceptions, dentistry, medicine, and pharmacy programs are de- 

centralized, and nursing and allied health pro'grams are 

regionalized, ' ' * 

) . It is felt that a regional effort to coordinate health man- 

"^.ower training/may improve Vhe capacity of health workers to 

^ operate as a /team, and may increase the ability of health care 

I* ,» 

pr^v^ders to ^W^T^^co'mmui^ity needs 

7 



I - ■ . • ■ ■ • 

'The creaMon' is an additional aspect of: the 

d/R HPEP concept^"- Nearly all decentralized and regionalized 
programs ^are established and operat^ed through .J. inks with local 
educational 'and health" car^ delivery ins t^itutions . Through 
these .linkages , it is hoped that a long run capacity will :be 
developed to assess the health care and health manpower needs . 
of the . community and to plan through education and other pro— . 
grams to meet these, needs. 

Types of Decentralized/Regipnalrzed Health Professional Edu - 
cation Programs ' . 

^ There are numerous hea^lth manpower education programs in 
existence which embody in varying degrees the D/R HPEP concept 
described above. Some of these programs were created by federal 

m 

programs. Some were initiated by medical schools, by local pr^^ 
vider groups, or by State agencies. Some of the long-es tabl ished 
programs have attained federal support recently.. Othei^s continue 



to operate independently^ o f federal programs. * 

Three main type? of D/R HPEP can be identified. One type 
is the Area Health Education Centers (AHECs) sponsored by the 
^ureau^Df Health Manpower. Under the BHM AHEC program, eleven 
medical schools have contracted with BHM to^d^velop remgte site 
training- programs . To,' date there are 29 AHECs associated with 
these medical schools. Many of these projects emphasize decen- 
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■ trairzation; of ' pHysician,' eciucatr BHM AHlEC projects 

,are also "involved in activities relating to .coordination of ^ 

* " • ' ' ' \ . 

allied health education programs and creation of linkages with- • ' 

community providers . * . . ' , <i> 

A second. type of D/R HPEP, known as Health Services/Education 

Activities (HS/E'^) is sponsored by the Regional Medical Rro- 

gram,. These projects, numbering approximately 85, -are independent, 

community based consortia of providers of health seryices and V 

' Uroviders of education and training. The' emphasi^s of many of 

these projects is on centralization at the. local level ofltallied 

heal'th education programs.^ HS/EAs represent an attempt to bring 

educational activities closer to practice by coordinating and 

• j ' . : 

supplementing' 'the clinical experience of students of member 
•schools. The basic philosophy of the HS/EA projects differs 
" f-rom the BHM AHEC concept in. that HS/EAs do not accord a . .. 

central position to the medical school or UHSC. Some HS;{EAs are. 
affiliated wilti' medical schools, but many are not. 

Veterans ^A.dminis tration hc^spitals conduct health pro fess io'nal 
education programs- which focus on creating linkages between 
education programs ancj^loc'al 4)rovider ins titutions^ Through 
these programs, the VA provides facrltlties 'for the clinical 
training of over 72,000 students a"yeaf. Local VA hospitals 
serve as catalysts for these acti vi ties , -vencouraging maxiiii^^ 
•of^heir facilities. All VA hospitals participating-.in- -D/R 





*A detaiNLed description af the activities of the 23 BHM AHECs is 
currently beiil^ prepared by Abt Associates, Cambridge, Mass 

'9 
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HPEP belong to local .consortia which helps to. ensure that their - 
e ffortsN^late to conditions and requirements oF local are^s . 

In additi.on to these three main types of D/R HPEP, a number 
of other decentralized or regionalized /lealth manpower training 
programs i^xist. For example, Michigan State University has 
deveTope-d a medical education program strongly emphasizing pri- 
mary medical care. In the absence of a'university hospital, all 
clinical training of edical students takes place in community 
hospitals in surrounding communities . The WAMI program at the 
University of Washington provides remote area clinical training 
for medical students -in Washington, Alaska, Montana, and Ida.ho. 
The Southeast Tennessee Area Health Education Center, funded by 
a number of sources, is a consortia of educational institutions 
built on the RMP HS/EA model. This program focuses on allied 
health Jnanpower training and is affiliated with several large 
medicffl schools outside of its region. , 

Goals and Objectives of Decentralized/Regionaliz e d Healt]^^^ Pro - 
fessi ona l Education Programs ^ ""'A 

• <■ 
Deftentrali zed and regionalized health professional education 



■v^ programs have been described 'as programs designed to affect the 
di/t^ib^tion of health manpower through changes in the educational 
process involving inter- institutional systems. This description 
provides a useful framework for s tat ing.. goals : one major set 
o'f goals pertains speci f ically ,to th.^geogxa^hic and specialty 
clis tribution of heralth workers*', a s.econd' is concerj>/d with the 
educational process, and a third relates to th^ organization of 
and' interaction among i^titutional systems. 



m 
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A stateiuent of f^/R FIPEP goals , subgoa Is , and objectives 
is outLined below. This list represents a consolidation of the 

r ■ 

views of the Advisory GrouQ .on Evaluation of D/R HPEP. 'It 
sh6uld be kept in mind that while each of the objectives cited ^ 
applies to at least one D/R HPEP, no single {iroject or program 
encompasses all of 'the objectives. As- indicated in the discussion, 
of types- of D/R HPEPs , there is a wide diversity across programs. 
The BHM, RMP, and VA programs were intentionally very loosely 
structured so that individual projects could develop innovative 
ways to respj^n^d to , local needs. Thus, pro-jects have established 
their own priorities and objectives within the framework of- a | 
national program. TlTB following list of goals and objectives 
covers most areas of concern to the various D/R HPEPs. 
Goal: Improve the (^stribution 'o.f health manpower resources.^. 



I 



Subgoal : Improve the , geographic distribution of ,healt|||^ 
manpower ' • ' 

Objective: Instill more Favorable attitudes toward 

\ remote area practice among medical stu- 

. dents. 

Objective:: Induce medical students to practice in 

* medically .underserved areas . 

• - " *^ 
Ob j ective : Provide educational activities for 

*: > medical students in medically under-, 

served areas . 

Objective: Provide clinical t^raining for medical 

students in medically underserved areas 

Objective: Induce established practitioners to ' 
practice in^pmedically underserved areas. 
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^ Objective: Encourage migration into MUAs . 

\ Objective: Di^coura^e ^migration out of MUAs. » 

Objective: Increase the supply of allied health 

workers in medicaiiy underserved areas. 

Subg.oal: Enrich the local professional environment. 

Objective: Establish con'tinuing edu^s^ion programs . 

^ Ob j ecti ye : Establish channels for consultation ^ an^ 

referrals between UHSCs and community 

hospitals. • ' - 

' *• » 

Objective:' Involve interested local practitioners 
in prpgram activities 

Stibgoal: Improve the t:5pecialty distri^Hfion o£ health 
manpower . * ' i,( ^ 

Objective: Induce medical students to .choose 'primary 
care specialties. 

•Objective: Increase the supply of primary care allied 
, health workers. - ^ 

Improve the educational process. • ' - ' 

Subgoal: Maintain or ^improve the quality of education as 
compared with traditional (central sitel edu- - 
cation program's^. \ 

< ' \ • ' 

.Objective: ^Assess local health manpower requrjrements and 

"determine the appropriate educational response 

Objective: Establish linkages and initiate planning 
among educational institutions. 

Subgoal: Modify organisational arrangements to extend ^ 
primary care education at all levels. ^ 

Objective: Estalrlish educational and. clinical training 
: V programs^in medically underserved areas 
V ^ 'emphasizing primary care. 



"'v^ObJ ec ti've -.Effect Specific cha'nges .in the. Hniversity; 
• \ / > Health Scie,nce Center .to : 

;^;> • Increase .-awareness df local problems^ 

•n.^ . - > among UHSC, [faculty and acimiais tration, . 

- Mqctify admissiohs policies to reeruit . 
.'^ ' more s tudents from rural • environments • 

Achiev^'' increased budget allocations to 
\s priinary care specialties. 



- 'Achreve expanded empha^^^. and curriculum 
^ ' ' ^ r /' content on primary cafe, 

. ■ " ' ' • Obtain.a long t.erm conunitment from the 

. • UHSC for contiWing Support of- D/R HPEPs 
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Subgoal: Modify education program content to improve -and 
increase heal th. education activity. 

Objective; Promote continuing education of health [ 
practitioners to update knowledge and 
upgrajie' skills* ^ 



Objective r Coordinate the education and training 

activities o*f all health manpower typ^es. 

Objective: Promote the adoption of a health team < / 
^ approach to increase, health manpower I 

^ product! vity . V. ' \ 

Subgoal: Modify the mix of .^udents entering educational 
programs . ' . 

Subgoal: Increase consumer health education! \ 

Subgoal: Increase the efficiency of the health manpower, 
education proces^s- . 

-^"^ " Objective: Use underutilized facilities for clinical 
■•■ . ■ • training.^* ' *' 

< ^ Objective:^ Utilize qualified local practitioners to 

supervise precep torships . 

Goal: Improve cooperative actions among i^n^tjl tut ional systems. 

Subgoal: Esta^blish inter -ins titutiSnal linkages to foster 
^communication? among groups in order to better 
relat^e health manpower training to local ne^ds, 
and in order to j^uild a system network which 
will promote the longevity of the decentralized 
or regionalized program, 

10 
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Objective: 'Create l-inka'ges among edudational institu- ^ 
tionsj and between education and provider . 
^' J groups. • /i*. 

. / ' ^ • ^ • / . 

. , - = Objective: Hncfourage informed -consumer vparticifeation 

\ * in community health affairs-. ' 
'A / ' ■ 

4 ^ AclgiviUes of .DecentralizedA-Regionalized Hj j ^th Professional 
Educat ion. Pr ograms ^ v^^ ■ - ^ 

. ' ' * V- • ' ' ' ; 

' Girven th-e^wide diversity which exists in program types and 
* . / ^ , ' • ■ ' . . 

objec'tives., it follows that the activities undertal<fen by various 

D/R HPEPs to/'achieve' their objectives also vary widely. In thi-sT 

discussion,- act ivi tie/^'are not lihked directly 'to '^bjecfeives 

sinQe activities o^ften serve multiple objectives'. Rather, several 

broad types of efforts are identified and examples of particular . 

activities, presented. ^e activities described are representative 

of the efforts of various D/R HPEP. The list is not intended to 

be all-inclusive nor is it intended to fully describe any par- 

ticular decentralized or regionalized program. Each D/R HPEP is 

characterized by a unique set o£ activities flowing from project 

objectives, available resources, and local coh|itions.* 

Most/D/R HPEP activities are directly related to education 

. or training programs fgr heal t^ "manpower . Efforts range from . 

designing and conducting training programs to coordinating 

existing prograips, to recruiting students and faculty, to. planning 

\ 

for future heal t|i -manpower needs. Programs are concerned with 
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*As noted previously, a detailed description of the activities 
of ill BHM AHECs is currentl)Mpeing prepared- The activities 

'of each HS/EA in California a^ described in Program Guide : 

The Ca lifornia Network of Health Services Manpower EducatTon 
rons crtia ,' Charles H. White, et. al^. , (California Regional Medical 
Frogram, January 197 5. • ' ^ 
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the training of students in' many 'heal th . fields and with the ^ 
continuing eduCatipn of es ta?J|^lished practitioners. Mostp 
activities are carried out in conj unction with area e^^2^tTorr- . . 
and/or provider institutions. A number a)f specific activities • 
are^detailedc below. * v ' * * 

Many DZ-R HPEPs are inv^ved .in designii^^^or conducting- ^ 
educational programs for health care^er stu^^ts. These, programs 
may train students in medicine, 'de'ntistry, nurs ing pharmacy , 
health administration, or allied health/ profession^ . The)^ may 
be under gjadu^^ta or graduate level training prograjns. They may • 
be pilot d^ons trati)On projects or' on-going programs. 

' In 'ad^tion Xo curricular program activities, mos>>^^R , HPEPs' 
are concerned with establishing clinical placements for students ^ 

in the local area.^, (Zl^nical t>fafiting takes placei primarily in 

- ^ ^'J / ■ 

community hospitals (or in VA hospitals under'' VA AHEC programs), 

if 

but also extends to preceptorships for medical students with 
lo^al physicians. ' Hospital -based clinical training programs for 
medical students and graduates 'm^y include short term rotations 



in 'community hospi talsV-internships , residencies including 
family practice residencies, or other types of programs. 

A related set of activities involves coordin^ing and-pTro 
viding assistance to educational programs already in existence. 
D/R HPEPs work with*local schools which train health professionals, 
and establish linkages between medical schools ^or UHSCs and local 
school s . Activities may include curriculum development^or , 

12 ' 



coordination of program offer-^Ys for various manpower types in 
an afea^, D/R HPEPs may also g.ive guidance to community'^ colleges 
or comprehensive colleges in developing new allied health ^ 



training programs . ' ^ ^ «> 

Recruitment -of 'Students to health careers is an important 
activity of /many D/R HPEPs in- light* of their ob^j ect ives of pro- 
viding training and employment opportunities' 'to local residents 
and especially t(o minl^rities and Jthe economically- disadvantaged'. 
Recruitment Activities seek to*!^ncre;a^€ awareness in the community 
o£ health ' career^ l?r^ining oppiDftuiji.tie$- thr9ugh use of-brachures 
•and counseling teams which visit schools . In addition, some 

. ■■ ■ ' J 

'programs offer 'scholarship.s to needy students. 

-Ail important concern of* most programs is the provision, of 
continuing education for local health manpower. Program activit-ie? 
may include ^ssessing needs aiid establishinff^rioritieSv for the 
content of continuing education programs, c^^p&cting legtures 
and discussions to keep practitioners abreast of new developments, 

providing for inservice training , |j^porting medical Audits of 

\ . • . . 

health service delivery, or establishing L^rning Resources 

Centers in area hospitals or libraries. Care is taken to con- 

duct. these activities in cooperation witH local pro-fessional 

societies and an effgrt is made, to draw on the resources of the 



UHSC wire re possible .ti^-if-^^ 



A wide variety of D/R HPEP activities relates to planning 
for future health manpo\^r needs. Most programs seelc to better 
relate health mahpow^f training to local' health care delivery 

13- 
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needs. In crder^ta accomplish this^; prpgrams may conduct ^ 



formal ''needs assessment'' in the. coijmunity, or may work with 

•l,bcal health .planning agencies and provider representatives . 

The basic activities include acquiring arnd anal];^zing data and 

eifcouraging open communication among local >interest groups to 

identify community heeds. Data of interest 'in the planning pro- 
^ . , I . 

Cess may include current supply w,f ^jaealth manpower, projected 

demand for services and for manpower, an inventory of training" 
^ ■ ■ ' ^ . ' ••■5 ' ■ 

program's, an^ inf cPrjijLation of rifanpower utilization, including job 

performlance ^criteria and- barriers to more effect ive , lit ilization', 

A planning activity o£'-.p/R HPEPs is tcr translate these 'data and 

others. into-pr,es.criptions .for health manpower . training programs^ 

Some D/R HPEPs engage in evaluation of their own programs, which 
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further feeds into the planning .process . ^ 

. The activities described above ai*e concerned with y^e 
•training and education of health personnel,' In a separate line 
of effort, some D/R HPEP activities focus on health care con- 
sumers,'^ These ' act ivities include consumer health education and 
self-help instruction, and occas^ionally diagnostic screening 
services, ^ ' -» 

A basic concern of both decentralized and regionalized , 
education programs is the creation of linkages - a network of 
commurijication and coop^eration among local education, provider, 
and health planning groups;' and between local groups and regional 
UHSCs. ^ Althouah a large .amount of D/R HPEP effort contributes 
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to -tlie formation of 'these inter -ii^^titutional ^i^ ' ^ev 

activities are desiVfted toNdev-eloP linKages Rather > 

rhe system network evolves as oyR %EPs work ^itV; ^oU^^^^ 

■hospitals, and planning boards to^Hiprpye tr^ ^fiii^^ grams ^^.'^ 

\health planning. j/\.'> . . ' 



\ 
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' ■ EVALUATION OF DECENTRALIZED/REGIONALIZED 

HEALTH PROFESSIONAL. EDUCATION PROGRAMS - 

In Se^ion i of this paper, the basic conc^ts o£ decen- 
.tralized and regionalized health professional education pror 
. \grams (D./R HPEPs) are discussed, and the gials, objectives, 

and activities of the programs are described. . In this section, 
•the focus of the discussion is evaluation of D/R HPEPs. At . 

^- ' -i • X . . 

issue is whether or not to undertake a national evaluation ot 



D/R HPEPs at *this time; and if so, how t.o* proceed in 'de signing '^^'^ 

such an evaluation. A consideration of »'why evaluate D/R HPEPs?" 

. • , ^ f ■ ■ 

2 gives substance to the fir^*issue anfl focus to the second. 

The general reason for conducting a F^derallyvsppn^sored 

■' . ^ ' ^ - V ^ ' ^ 

national^ evaluation of D/R HPEPs can be started simply: to 
improve Fed^sral decisioji-making relating to eertajn health man- 
p^er issues. -Federal policy is ^now 'act rvely seeking througlii 'a 
: number -of mechanisms to affect the geographidjf&istribution of 
health manpower and to increase concern for'primary care. The 
Federal- Government ' s support, of D/R HREPoS 'may be viewedVas a 
form (if social experiment intended to _accom^lish these^oals. 

Although 'the scope M FedeYal involvement in '^)/R HPEPs is 
small in the context of Federal monies spent on health,. or even 
on health manpd'wer training, it is important j:o assess . the- per- ' . ■ \ 

I 

formance of these programs, at this time for several reasons. ' ' 

Decisions must be ni^ade regarding renewal of graiits o,i\c^ontract s 
supporting current programs. Because the programs are relatively 



16 

r 



19 



ERIC 




t 



new, there is no accumulated evidence' on their performance. 
Even knowledge of descriptive tharaclSerist ics of the programs 
is limited. ^Furthe.rmore , the^trate^'^is \i:|novative. The 



C 

effectiveness of intervening in the ^ucat^i}al p^ob^ss to 
influence location and Specialty 'decis]ions^s unkipw^ In*<-i^t, 



numerous reservations about this ^strat^^f have been expres 



r "7 

yen expressed 



in view of, the l^ge number 



o^ variables Which affect ti^ese / 



decisio^T;is . \x this point, it is-not clear how well conceived 
the strat-^y is or how severe -the barriers are to effecjt^ve 
implementation. * 

Decisions must be 'made .not , only on the appropriate level 
of sup^port for currently-sponsored pro jects ,^.p^|i/feM whether 
or not to ^expand these ,and::^^imi 



anc 
oVt 





AHECs and HS/EAs^^ere noVt explicitly e< , , 

tiori projects to lay the groundwork for a Laftge- scale national 

program. However, if they ap^p^&r to b« a riinSductive qjad effi- 

cient strategy for achieving national goals, there is reason 

to consider expanding^ Federal «a;nd o^^hej;;.^ A current' 

evaluation of D/R HPEPs is essential to -t/ie consideration of 

whether or not additional AHECs **or HS/EAs should be fdnded, and 

■V . ■ & 

if so, where and with what provisions. 

A second reason to evaluate b/R. HPEPs is to improve lo(^l- 

level detision-makinjg -regarding resource allocation and operation 

of specific projects. In some cases, part ic^ularly in BHM AHECs, 

a continuing eva:i,uat iqri. effort is 
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the project, ^owever, all projects, whether or, not th^y conduct 

. • - ). . . / r 

self -^aluation^ , might benefrt from a broader evaluation, ^ 

' V •> , ■ ^ . # - 

whiqh woulH^' provide informatipji on **best practices'' or most 

effective approaches in various ^ettin^s . Thus, the- response 
to ''why evaluate?" is straightforward: to provide information 
Xo^ Federal and local policyinakers ^n the ^rf ormance^^and impact 
of .these new, innovatiVe programs. . , ' 

"Th© issue of whether or ngt to undertake a national evalua- 
tion at' this time iV clearly too sjfi^ly drawn. V range of < 
options exists from conducting nQ evaluation thit^ugh increasingly 
more comprehensive, and cons^uently more costly and ti^e-con- 
suming, evaluat io^^^^eff^ The decision to b^jilade is one of 

de'sired and feasible scope evaluatl^r^ 

^ This decision will ultimately be Wade by the Federal 
sponsor off the evaluation, in ligkt, of '^^xist irfg constraint's* 
relating yoltime, budJget, and availability of data. The anti- 
cipated aenevitjs of the information purchased, to the sponsoring 



agency to oqier intere|||^ed parties, must be weighed agai-nst 

tKe"^"eWimat^^ As input into this decision, the present 



discussion examines alt^narives fe lated to the evaluation 
desigrji^and considers certain problems which will be encountered 
in evaluating D/R HPEPs. These problems relate Xo the availa- 
bilityVof data, the length of time D/R HPEPs have been in 
operation, the diversity of project characteristics and settings, 
and mea^surement -and -attributian o-£^^^ impacts 

21 



Any limitations on the scope of the evaluation which might 
derive from time or budget constraints faced by the spoi^soring 
agency * are '^i'gnored at this stage. Speci fiq topics which are 
cons idered , include : 

• Objectives of the Evaiuartion 

I / For whom is the evaluation to be ^conducted * 

and what are their policy, concerns? ^ /-^"""^ 

What broad policy concerns are genera^e^^ji-^ 
'\ D/R H'PEPs? .^'^ . 

' ^ What types of evaluation might be conducted? 

• Targets of the Evaluation 

<^ ' What programs are to.be studied? 

•nr What particular projects or sites are to be 
studied? 

• Subjepts for Evaluation 

^ What types of information are desir^ed? 

What dimensions, of :t)/R HPEP activity^are to ' . 

\be assessed? - ^ 
Potential evaluation questions . ^ 

• Measures 

Problems in Evalu^ion^ of D/R HPEPs 
Objectives of tn^ Evaluation • ^ ( 

It was suggested above that the basip rationale for evalu- 
ating D/R HPEPs is to improve certain aspects of ' FederalV^and 
local decision-making. To tran^^^a^e this general concern into 
directives for an evaluation desi^n^^H?i,.J,s necessary to identi 
dec i s i on - m aking group s^^ D/R HPEP performance and. 
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impact and their policy concerns^ and to recognize broad 
policy concei:ns generated by D/R HPEPs Corresponding to 




ithese policy interests, Several basic types o£\ evaluation whiph 
might be conducte(^ are described below ^ 

Policymakers and administrators interested in l//R HPEPs 
include the Congi^ss', the Office of Nfanagement and Budget, 

^ Fe'deral administrators ' ^n the Office of the Secretary of DHEW, 
in DHEW*s Health Resout*cep Administration, and in the Veterans 
Administration, progranv cbordindtors in the Bureau -of Healt^^ 
Manpower, the Regional Medical Prograins, and the Veterans ^ ^ 
Administration, and project adminis tTa^rs at the'^local level 
State legislatures and o^-er.^/R HPEP funding sources an^ area 
education and healfh planning groups also are likely to be con- 
cerned with the performance of D/R HPEP'^'t however their specific 

-policy concerns will not be considered separately here. 

Congress ,r\Concerned primarily with the effectiveness of 
Fed§Wj^^programs in achieving social goals, must make decisions 
on ^^tur^ ■ authorizing legislation and allocation of funds to 
Federa]yJ|^sponsor»Q programs. It will be interested in the 
achievement^^C^M, RMP, and VA D/R HPEPs in relation t^ 
national goals. J \ 

0MB is charged with assessing pyerall Federal resource 
allocation to assure that Federal programs are run effectively 
and efficiently. Thus, 0MB staff will^want to know if Federal 
funds are being used for the intended purposes and if program 
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achievements are reasonable in light of resoifrces expended. . 
Evaluative information relevant to these concerns includes a 
compar,ison'-o£ funded activitie-s wi^th program directive? and 

r • ' ' ■ . N, ■ 

goals, \ and a co'jnparison of achievements with costs. . 

Fe\ieral administrators at various levels focus^ attention 
on allocation of Federal resources for health and on program 
development. Decisions must be made on whether or not to con- 
tinue support of BHM, ;rmp, and VA programs. Rela't&d decisions 
concern how resources should be allocated among the^e pr.ograms 
and between D/R HPEPs and alternative intervention strategies*. 
For these decisions^ policymakers need information on the 
effectiveness and efficiency of Federally^Strpported D/R HPEPs 
and of other Federal' p|^grams which may serve the same ends. 
In order to make decisions related to -^o gram development, 
information is needed on the relative effectiveness of various 
approaches (e . g r/^decentralization, regionalization) within the 
D/R HPEP strategy. 

A broader concern o£ Federal administrators and the Congress 
which bears on resource allocation decisiofhs is what is the 
appropriate role of the Federal Government' in D/R HPEP develop- 
ilient? What difference has Federal support and establishment of 
a national program mad'e in the development of D/R HPEPs and iiT«- 
the achievement of basic goals? Has the linkage network^ pro- 
moted in Federally- supported decentralized, and regionalized 
progr.ams made these programs more effective than isolated. 
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remote site training programs in^achieving basic goals? *To ^' 
address ^these issues requires information on achievements of 

i . ^ • 

Fede'rally-sponsored D/R HPEPs , non- Federallv-sponsored- D/R HPEPs 
and isolated remote site training, programs . , * > 

"Program coordinators are co;^cerne*d primarily with program ' 
development' and managen\ent . They need to know hW well tHeir 

i 

programs are doing and wh,^ structural and management changes. 
to promote, to Improve program performance 'and increase ef^i- 
(zj^ncx^. A review of proj'ect^achievements an'^-of %he process-- 
what implementation strategie's' are most effective-.-is requited: 
for this assessment- 



Project administrators, are a^lso concerned with program 



develo|)ment and management, but on a local pro ject,<X^^;^^ ... . 
Managemelit decisions concerning ^ the appropriate . level and mix - 
of staff, appropriate lines *o^|plithor i ty , and dfegree of monitor- 



ing are required. Resource, allocation decisions for program 
■content must also be made. In orcT^r^^^ ^improA/:e.^per f ormance , 
project administrators need inf oxmat ioiT^n their^roj ect ' s 



achievements and the xredat ive effectiveness- of varlod'l spon 



Bored activities and staff configurations. Informat 
jjSother pi::e^ects in similar settings will broaden the local 
administrator ' s awareness of ways to improve hil project's' 
performance . . 

"Primary users" of the eValtiation must be chosen from 
these groups, who wjll participate in defining what is to be 
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evaluated and what constitutes acceptable evidence. The 
desigji'ation' o£ primary users does not prr^eclude others, from 
using tl^e information agqui-red. •'Clearly, policy concerns of 
the vafiifbus groups overlap and information produce'd^or one ^ 
group 'may be of /interest to others. The selection of any pfii- 
mary user,s, in addition to the one or t\vo most^important groups, 
sjiould be based on benefit-cost considera1:ions , weighing the 
availability of valid information, and the costs of acquiring 
additional^inf ormat ion (and. of cor^tending with additional- par-> , 
ticipahtiS -in 'the definition of evaluatioit- issues, and acceptabl^^ 
evidence)'^ against the benefits to-:\tKe sponsor of the^ evaluation , 

Several' broad policV JtssUes can be ioTem'tif ied frora^the 

- ■ i'' ■■ * 

concerns des(A'^ibed"al)Ove First , how .effect ive has^the B/R' HPEP 
Svtrategy beeji in achieving or mbVing the nation> .toward specif ied 
goats and objectives? How efficient have the programs been?^ 

... V 

Ji(hat factors appear^to'- improve or to hinder siI5^ss£ul per- 
formance? And filially,- what appears to be the mon appropriate 
role for Federal support of D/R HPEPs? 

The first issues can be addressed through a benefit-cost 
assessment of D/R tIPEPs, focussing on tke basic questions: 
1) What progress has been made by D/R HPEPs toward achieving 
^oals and objectives related to the geographic aad^jpecial ty 
distribution of health manpower, to the education and training^ 
process, .and to the fostering of inter- ins t itutional systems?; 
^and 2) what is the net cost of D/R HPEP operation? 



In assessing benefits, attention should be given to 
unanticipated program achievements which may or may not contri- 
bute to achieving, objectives. Benefits to whom? and along what 
dimensions? must be defined, since these are mlilti-'dimensional 
prograjns :and/^^here are many parties , with vested interests in 
outcomes • - ' ■ . " ' ■ v 

An assessment of D/R HPEP costs must address questions 
such as: What are the net costs to the Federal Government of 
D/R HPEP support? (What other Federal fund allocations are 
lessened as a result of D/R HPEP?)^^' What is* the distributional 
impact of D/R HPEP support as opposed to alternative program 
support (NCHS, Family Practice- Residency Program, scholarships 
and loans")? The costs to whom must be speci*fied, (Possibilities 
include Federal and State governments, local communities, 
students, society,. 

The third concern, identifying possible reasons for. program 
success, is relevant to' all policymakers seeking to improve 
program p.§rf ormance , Due to wide project heterogeneity this 
assessment must be conducted with respect to particular com- 
ponents of D/R HPEP activity. Nevertheless, general project 
character is tics such as organizational structure , management 
factors, operational age and_ his tory of ' the project, and the 
receptivity of the local environment to planned changes should 
be considered as well as specific activities undertaken by 
projects to achieve objectives. * 
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Concerti about the /Appropriate Federal role in D/R HPEPs 
encompasses a broad range o£^ issues. Specific questions o£ 
interest include:- Are there systematic differences in program 
achievements between Federally-funded programs and programs 
funded from otheT sources differences which can be attributed 
to source^of funds? Does i^^^^^^^ that' Federal funds have 

been necessary for stimulating the development and/or expansion 
of D/R HPEPs? Are Federal funds being used for intended pur- 
poses? Have administrative regulations (e.g., decentralization 
of DHEW) hindered ,or facilitated the development of D/R HPEPs? 
What' appears to be the optimal level of Federal funding for D/R 
HPEPs? The optimal number of projects in each program? The 
most reasonable time period for-Federal support? Most of these 
issues axe quite difficult to assess and will require a large 
decree of judgment in interpreting information from programs 
which-' are Federally supported and those which are not. 

From the, discussion of potential primary users and policy 
concerns, five general types of evaluation emerge: a national- 
program evaluation, a comparative evaluation of types of D/R 
HP^Ps , an evaluatidn of best practices for attaining a certain 
limited set of objectives, pro j ect - levej evaluations, and an . 
evaluation of the impact on geographic and specially distribu- 
tion of D/R HPEPs and alternative intervention strategies. 

A national program evaluation (for instance of BHM AHECs) 
woulfe entail examining all of the projects'in the program, if 
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p'ossibl^. ; The central concern is with the program's contri- 
bution to achievement of certain national goals... These goals 

might be those indicated in 1971-1972 at the outset of the 
programs, or tl^ey might be other goals of current or future 
interest. - 

■* a) 

A national comparative analysis of types of D/R HPEPs- 
could entail choosing a sample of projects from each program 
type of interest and comparing performance and impact. by pro- 
gram typ/i^. The main , parameter o£ interest "in this type of, 
evaluation is variation^n organizational structure and' philos- 
ophy, and the consequent effect^ on program achievements • , 

To'^onduct an evaluation of best practi/ces linked to 
specific objectives, evaluators would have to prioritize objec- 
tives and identify the few* of highest priority, choose for 
Evaluation only those projects specifically addressing trhose 
objectives, and evaluate alternative approaches for achieving 
the objectives., Management factors; as well as funded activities 
ai'e topics of investigation here. . " • * 

- Project level •evaluat'ians- would be concerned with the 
degree to which individual projects are meeting local ly-^assessed 
needs and achieving locally-defined objectives. Special atten- 
tion is to be directed to the historical development of* the 
project and to the political and economic context in'which it 
operates . , • . ^ 

' The final type of evaluation cited is that of identifying 

areas which have recently achieved a satisfactory supply and 
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Ijalance of health manpower types and assessing the relative 
..role... o.£. .D./il._HP.EPs ._ i n_ t h i s__achie vemen.t . .Th i s t yp e ; o J& j^va lua t ion 
is more complex than those described above in th^i it is ' 
neqessary to try. to account for the "influence 9^ the, manji factors 
other than '^D/R HPEP activities affecting practice location and 
specijalty choice. The general approach to this type of evalua- 
tion would entail comparing a large number of selected areas,- 
perhaps 30 sites, located near decentralized or regionalized 
training programs, with 30 additional sites similar in general 
characteristics, yet without D/R HPEPs. 

As suggested above, the choice of programs and specific 
project sites t(^^e examined , 'depends 'upon the type of evalua- 
tion conducted and the objectives of the evaluation, and is a 
matter for agreement among primary users. Key considerations 
are likeLy to be the number of activities of a given type 
reported to be underway and the potential availability of 
information,. Decentralized and regionalized programs which 
could be evaluated might include the 29 BflM AHECs , approxi- 
mately 85 JIMP. HS/EAs, . eight. VA AHECs,. unsuccessful, applicant? 

for BHM contracts and RMP g/rants, and various other related 

r 

I 

programs such as WAMI , Tennessee ^ s SETAHEC, WICHE, and others. 

In reference to the broad evaluation issues , the benefit- 
cost assessment described might entail comparisons within or 
across program type. Evaluation of alternative approaches to 
particular objectives as mentioned above, would benefit from 
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wide diversity ^mong program types' examined. . Finally an 
evaluation o£ the effectiveness of Federal funding would 
necessitate looking at both' pro jects' receiving and projects 
not receiving sizable Federal ^upport . (J 

In terms of the type's of ^devaluation discussed above, a 
national program evaluation would permit examination of projects 
_from a single program, whereas ^projects from alternative pro- 
grams would have to be included for an evaluation of D/R HPEP • 
types. An evaluation of besr'/|practices within the D/R HPEP 
strategy would be most benef icial if it were 'based on proj-ects 
drawn from a variety of programs in order to 'capture substantial 
variation in organization and management as well as particular 
activities. 

There exist three alternatives £or,chosing projects to be 
examined: the universe (all projects in a given program), a ^ 
random sample o£ projects from one or more programs, and a 
sample selected on the. basis o£ specified criteria. 

For a complete p^rogram evaluation, one would want to assess 
overall impact by. examining tlie performance of each sponsored 
project, unless, as is viikely with the RMP program, budget 
constraints and the unavailability of info'rmation. prohibit such 
an undertaking. To compare program types , a random sample of ^ 
projects from, each program might be most desirable. In evalua- 
ting alternative approaches to particular pbjectives, one would 
want to select only those projects working toward th/ objec- 
tives of interest. Depending on the " ob j ect ives of the evalu- 
a-tion, factors such as the size of a project, the number of 

28 

31 



years in operation-, the sources o£ funding, and other character- 
istics, -may be relevant in selecting the sample,. 

Subjects , Fo r Fvaluation " ' 

^ ~J ^ ' . 

In order to clarify the information requirements described 

above, and to focus attention on particular evaluation issues, in 

thi$ section data about p^^^ect performance are classified into 

four types of information. Subsequently seyeral dimensions of 

D/R HPEP activity related to major goals and objectives,, are 

identified, A number of potential evaluation questions in each 

of these dimensions are then proposed, 

' Information necessary for evaluation of D/R H^P per- 

formance and impact can be described in terms of the following 
categories: ' ^^^^^ 

^ ' 1 

m Achievements 

• Costs and Source of Funding Support 

• Activities 

• Characteristics of organizational ' 
structure, management, and project 

^ setting. 

The first two types of information pertain to program or project 
inputs . arid outcomes, The second two relate to the process of 
achieving outputs,. Any assessment of the effectiveness of project 
activities must be accompanied and qualified by consideration of 
the other (endogenous and exogenous) factors mentioned. 

Explicit consideration of time is necessary in determining ^ 

s 

information to be obtained', on both achievements and processes, • 
Different outputs and" especially impacts are to be expect-ed in 
different time frames. In .the context of D/R HPEPs , the first 
one or two years of operation repres%t a "short-run" period. 
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Three to four years and 7-10., years of operation nyight be thou'ght 
of .as intermediate and long-run periods resplsctiyely . The entire 
life span of a project is a more relevant time reference than 
the length of time it has been part of a particular program. A 
fully operating project which acquired BHM funding in 1972 
should not be compared directly in terms of achievement with a 
BHM AHEC which was initiated in 1972. 

6 

The general classification scheme presented abo^e draws 
attention to the broad types' of information of interest "^in an 
evaluation. More specific to the evaluation of DR/HPEPs, there 
^re several dimensions of project activity relating to goals and 
objectives which might be subjects of evaluation. It will be 
recalled from the discussion in Section 1 that D/R HPEP goals fall 
into three' broad areas. The programs seek to influence the 
geographic and specialty distribution of health manpower by 
effecting changes' in the educational process and in the process 
building inter- ins ti tut ional linkages. An additional, explicit 
goal of some programs is to create employment and education 
opportunities for minorities. 

These four goals of DR/HP^EPs can be regarded as possible 
dimensions to be evaluated. The dimensions roughly correspond 
to. a time flow, with efforts , in building intper- ins t i tut ional 
s^ystems supporting efforts to change the educational process, 
which contribute to .changing the g^eographic and specialty 
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distribution of health manpower and ulti^mately improving the 
availability and quality of ca^^e .ift D/R^HPEP ar^as . All dim^nsi^s 
develop over time, but in terms of a:cu'rrent evalfation, outputs 
are more likely to have been achieA/-ed in the areas Vof institutional 
linkages and educational process changes. While impact on 
geographic-and specialty distribution of health manpower may be 
the primary evaluation concern, this dimension of an evaluation 
wyuld have to rely more on process 'than' output measures at the 
present time., '' ' - 

Improvements io^ the quantity and quality of health care in 
medically underserved are§s are the ultimate goals of D/R HPEPs , 
However, considering the multitude of factors which influence 
the availability and quality of care, and considering the r<^la- 
tively small role piaved by D/R [iPEPs in the total health care en- 
vironment, these dimensions are not considered here explicitly 
in r^ atlon to the evaluation of D/R HPEP performance. It is 
■ assumed that desired impacts on the geographic and specialty^ 
distribution of health personnel will bring about impr.ovementi 
in health care delivery, 

A number of potential evaluation questions linked to these 
dimensions of D/R HPEPs are set out below. Most of these questions 
are framed in terms 'of outputs or impacts, but information on 
costs and on the process of achieving each objective may also 
be ^desired,' The particular evaluation questions to be studied 
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within *e4eh\ dimension will depend in part an the actual amoufit 

\ , 

of activity being conducted. Information from the current study 
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by Abt Associates, Inc. will indicate relevant areas of BHM AHEC 

^ I; / ' ^ 

activity'' to be evaluated. 

Potent i^^f^'^aluation Questions . " ^' ' * 

^ ■ r ' 'A 

. Institut.ipnal System Building 

• Have meaningful linkages been ^stablished amopg UHSCs , 

local health manpower educatixrn ins t itutions ^ and 
local treatment facilities ? 

Have meaningful 1 inkages . been established between 
f 

these education and provider institutions and area . 
health planning agencies? ,^ ^ — 

Have local providers individually been drawn into 
D/R HPEP activities.? 

What is the app^ent degree of committment of involved 
medical schools and allied health training schools ^ 
to ^the development of the remote site training pro- 

V. ? gram they are associated with? 

■ - • V- - . T . Has --^the project contributed .to informed .consumer 

participation in health related affairs? 

• Educational Process 

Has the quality of education of traditional programs 
been preserved in remote site traihing programs? 
Has it improved? * 
- How does the content (curriculum, clinical training, 
relative emphasis) of D/R HPEP training programs 
differ from traditional programs? 
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In what ways or to what extent do changes in training 

program content and emphasis serye to better relate 

training programs to "^local neeAs? 

What is the impact of D/R'HPEPs on the< af f il iated 

(if any) UHSC or medical school itself in terms of: 

increased awareness of local problems among 

UHSC faculty and administrators? 

♦ ■ -. - 

recruitment and admissions policies? 

allocated budgets for primary care specialty 

training? : - - ^ 

' curriculum content and emphasisre^ted to 

primary care? " ^ 

commitment of UHSC faculty and administrators 

to continuing operations and development of ^ ^ 

D/R HPEPs? 

Have meaningful continuing education opportunities 
been established and utilized by local health man- 
power? 

Distribution of Health Manpower 

- Has the D/R HPEP ' influenced student attitudes 
(favorably or unfavorably) toward , pract ice in 
medically underserved areas (MUAs)? 

-- what is the independent influence of the D/R 
HPEP experience apart from predisposition 
(self-selection bias), on attitudes of students 
participating in D/R HPEPs? • ^ 

, A . /» 
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1 ' 

what is the impact, if any, on attitudes of 
, students not participating directly in D/R 
HPEPs? ' ^ • « 

^ -i-- do attitudes differ toward practicing in MUAs 
surrounding remote site training programs as 
opposed to MUAsL without near-by training pro- 
grams? * . 

- Have students ^participating in- D/R HPEPs been induced 
to remain to practice in the surrounding area? Are 
these location decisions attributable to D/R HPEP 
training program content, or to the location of 
training or to other factors? 

- Have established practitioners been induced*to .m^igrate 
to or remain in areas surrounding remote site -training 
programs? " 

--gl'what evidence is there of an improved local 
professional environment? 

- Have medical and allied health students been induced 
to choose primary care specialties?. ' 

- Have established practitioners (physicians and allied ( 
health personnel) been induced to spend more time 
delivering primary care as opposed to specialt)^i. care*? 

Education and Employment Opportunities of* Minorities 
Is there an increased awareness of education and- 
employment opportunities among minority residents in 
D/R HPEP areas? 
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- Has the proportion '^of minority students in health 
• career training program^ increased? 

- Has the proportion of minorities employed in health 
careers in all locations increased? 



Measures 



In the present context of consi'dering many alternative evalu- 
ation objectives, primary users, and information requirements, the. 
specific measures to be used and data to be collected cannot b'e 
considered in detail. In general, certain accommodations arid 

narrowing of scope may be necessary if desired data are -not avail- 

/ 

able. Several other general considerations need to be kept iA / 
mind. . * 

# 

There should be agreement among all primary users that the 
data collected in relation to each evaluation rquestji^|i -iTep-resent 
acceptable evidence." This is especially imft^fant wbete phenomena 
of interest are not directly observable and must be ;measured by 
proxy variables. 

The quality or effectiveness of HPEP activities is 

important in choosing measures and determ,ining da^^'to be vcoldected 
For instance, the number of linkages ntfeans lit-ile without knowledge 
of the degree of institutional committment behind ^he linkage. 
The number of continuing education sessions reveals little without 
knowledge of content, correspondence to need, and participation. 
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The above references to data ^requirements and types of infor.- " . 
mation involve five kinds of measures including context, input, 
process, output and impact measures. Context measures pertain 
to general characteristics of. the programs such as thei,r organi- 
'zational structure and management chairacteristicsT and to th-e 
political, economic, and socio -demographic characteristics of 
. ^ the "host" community. 4 

Inputs refers both to dollars flowing thro.ugh the project 
and to physical resource such as faculty, administrators, hospitals, 
and classrooms. Process measures describe project activities - 
the ways in which resources are combined in order to achieve 
objectives. Examples might include the number of Family Practiced 
Residency positions made available by the project or revisions ^ f 

in UHSC curriculum incorporated in remote site training programs. 

Output measures describe the results of D/R HPEP activities 
and are defined in relation to project objectives. Outputs can 
be defined at different levels of speciAcity: a sub -obj ect i ve 
might-be to expose a^ number of students to a remote site clinical 
experience, measured by the number of students' participating 
during a given time period in a specific clinical training pro- 
gram. A related objective might be to induce these students to 
choose a primary care specialty or to chopse to practice near the 
^ dec-entralized or regionalized program, measured by the number of 
students who were influenced to do so by tbfeir experience in the 
D/R HP-EP. . / 
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" While output measures relate to project objectives, impact 

.ft ■ 

measures s^eek to assess the effect of project outputs on goal 
attainment in the context of a larger environment where factors 
outside the sphere of the D/R .HPEP, are operative. 
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* The ^five kinds of measures described here correspond to the 

types of information discussed above in the following manner. 

Achievements are measured in terms of outputs and impacts. Costs 
: are measured in terms of inputs'. Efficiency is measured through 
ja comparison of outputs or impacts and inputs. Aqtivities are 

described, as noted, by process measures. Finally, the additional 
^f actors affecting performance, both endogenous and exogenous 

factors, are measured by context variables. 

Measurement problems arise to a greater or lesser extent in 
all of these categories. Measurement of changes in student 
attitudes toward practice in medically underserved areas, a sub- 
objective of D/R HPEPs, is elusive. However, attitude scales 
have been developed and can be employed to give some indication. 
...>Ieasurement..pf.,context_ V the most problematic . 

In terms of management factors, j.t is not evident how charismatic 
leadership or institutional 'committment are to be njeasured. 
Further, out of the multitude of political^ .economic, and socio- 
demographic characteristics of the local s&tting, it is not ^ 
obvious which characteristics or what aspects are immediately 
relevant to the performance of D/R HPEPs or how to measure them. 
Furthermore, the decree Oji influence of context variables may 

Jl 
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fluctuate during the lifetime o£ the project. Clearly, there are 
no rules of thumb for characterizing the influence of context^ 
variables. As stated at the outset ot this section, these and other 
measurement definitions and problems must be dealt with within 
the framework of a specific evaluation design. 

♦ . 

Probletns in Evaluating D/R HPEPs ' . 

* • (• ' 

Several factors in addition to measurement dif f ictriKies 
mentioned above create problems for, evaluating D/R HPEPs, factors, 
which on one hand limit ,the feasibility of evaluating certain 
aspects of the programs, and which on the other hand place 
special r^uirements on t^e evaluation design. These factors 
can be described in terms of delayed impacts, data availability 
and comparability, attribution problems, 'and project hetergeneity . 

Delayed Impacts 
Delayed impacts simply refers to the fact^ that the overall 
goals of D/R HPEPs are long-run goals which will require a number 
of'years to attain. The training programs can ' certai ^ly be 
evaluated now. However, i^nless the timespan of . the^ evaluation 
can be extended to cover the relevant time period perhaps .the 
next five to eight years a current evaluation must focus on 
evaluating processes and production of interiftediate outputs. In 
this regard, the qualitative dimensions of project measures are 
^specially important. The probability of future "^uccess" depends 
not only on the existenc4 of various activities, but- also on their 

• . \ . ■ . ■ • 
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appropriateness for meeting local needs (and national ;goals) , and 
the extent to which they build in a long term capacity for continued 
efforts- ^ 

^ Data Availability and Comparability 

The efforts of Abt Associates, Inc, to describe BHM AHECs 
have revealed many problems related to the availability and com- 
parability of data from different projects. They observe:* 

The projects with which this evaluation will deal 
probably vary tremendously with regard to the types 
of internal information they collect and how they 
record it. Some projects will be found to have 
detailed program budgeting and management reporting 
systems (keyed ,to activity milestones and budgets) ; 
others will have trouble in listing their various 
program efforts or in generating program cost data 
at all. In some projects, basic program activity 
data will be maintained 'at one central lacation 
(sometimes in computerized form); in other projects, 
this information will be maintained at various remote 
sites, within individual participating institutions 
(who may have never previously been asked to report it 
to anyone). Some projects' periodic reports to 
sponsoring agencies will provide a useful picture of 
program operations; in other cases, spending a great 

4eal of time on complex narrative reports -can~-confuse 

the observer more than it informs him. Finally* there * 
is the problem of project staff turnover since the 
start- of the project. In a pro j ect setting where 
^individuals (instead of systems) keep track of informa- 
tion, such turnover (particularly if several top project 
staff are affected) can cause a serious gap in the 
record of project activities, ^ i 

In anticipation of these problems, the evaluation design should 
incorporate data collection protocols which are flexible enough 
to obtain the maximum possible amount of desired data. If certain 
data items are not available, other indicators might be sought. 
In cases where no data are obtainable or where data reporting systems 



'*Paul Grigorieff, ''Evaluation Concepts for Decentralized Health 
Education Programs,'* Abt Associates, Inc, August 1975 , p, 17, 
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do not permit cross -proj ect comparisons on specific items, certain 
evaluation questions njay have to be omitted from consideration. 
• * Attribution Problems 

Given the complexity of the e^nvironment in \vhich D/R.HPEPs 
operate, it will be exceedingly difficult to identify impacts of 
the programs, that is to be able to attribute observed changes to I 
HPEP efforts. Clearly there are many factors, including some out- 
side of the sphere of D/R HPEPs, which influence the geographic 
distribution and specialty choice of health professions. With so. 
many influencing variables, so few projects and no formal control 
groups, it is impossible to establish causality with certainty. 
If suitable control areas can be found for comparison, they may 
provide the basis for informed judgements regarding the impacts of 
D/R HPEPs^^ 

A further attribution problem exists in trying to assign 
responsibility for outputs to specific project components. This 
problem arises because many of the projects are "open systems','' 
involving dif f erent organi zat ions and activities. -Thus it is 
difficult to identify specific ins^tutional or program responsi- 
-bility for some achievements. Also many activities overlap in 
purpose, contrUbut.ing to the achievement of several objectives. 
This problem is encountered only in the evaluation of processes or 
best p^-actices for achieving, obj ect ives . 

Pro j ect Heterogeneity 

The extensive variation which exists in project description, 
including objectives, activities, and organizational structure, 

40 ^' 
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poses problems for evaluation design. The interests of policy 
makers, as described above, encompass both program performance and 

■ project performance. However, there is a trade-off between a case 
study approach, which captures the unique characteristics of 

,^ individual projects, and a program level evaluation, which generalizes 
about the performance of all projects taken together, 

A program level assessment cannot accommodate all of the 
^variability in characteristics and settings of individual projects 

; which must be ignored or subsumed in broad generalizations, 
At the same time it is likely that some projects may be evaluated 
in terms of objectives of national interest which they themselves 
have accepted, or to which they have assigned low priority. This 
is perfectly legitimate for a national program evaluation, but is 
likely to generate little enthusiasm at the project level. Since 
project cooperation is necessary for a successful program evaluation, 
this point may have implications for the evalua^on design. A 
reasonable accommodation would be to retain^ the nationally important 
/^objectives as grounds for evaluation but expand the evaluation^ con- 
cerns to include the atypical activities a;nd efforts of various 
projects. If this method is carried to extreme, the result is a 
series of case studies where the quantity of information becomes 
unmanageable and the ability to generalize about program performance 
is diminished.' 

Also because. the projects and their settings are so heterogeneous, . 
^effects of project operations are not predictable. The evaluation 
design should be flexible enough to capture unant ic ipated . as well 
as anticipated consequences of D/R- HPEPs. 
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Scope and Level of Effort of the Evaluation 

Returning to the initial issue of concervn, it ^an be seen that 
in light of the broad range of potential policy concerns, targets 
and subjects for evaluation, different types of evaluation can be 
conducted and the scope and focus of the evaluatioii effort (how many 
and which evaluation questions are addressed) sre variable. One 
option mentioned at the outset is to conduct no formal evaluation. 
This would be a reasonable • course of action if it appeared that 
^evaluation results at any level would be too inconclusive to merit 
the necessary investment or that time constraints precluded a mean- 
ingful evaluation. This option might be combined with a decision 
to increase efforts to institute a uniform management information 
system, facilitating future evaluation efforts and creating the 
capacity for longitudinal anal/sis of program performance. 

A second option would be to conduct a program evaluation of t 
one national program. At least 15 months would be required to 
design and conduct the evaluation and interpret results, allowing 
time for current MIS efforts to yield initial results. This type 
of evaluation is roughly estimated to cost between $300,000 and 
$400,000.* 

As another alternative, the capacity to compare across program, 
types might be added to the program evaluation. The necessity of 



*This estimate corresprOnds to an evaluation of the Bureau of Health 
Manpower AHEC program. Because of the number of projects and sites 
involved, program evaluation of RMP HS/EAs and VA AHECs are likely 
to .cost more, and less, than this estimate , respectively . 
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examining projects from a variety of programs would raise , the over- 
air costs to perhaps $400,000 - $500,000 and require a timespan of 
approximately 18 months. • * - 

An in-depth evaluation concerned .with identifyir^^^'be practices 
in achieving a few specified objectives would add approximately' 
$250,000 to $350,000 to overall costs and would also require in 
total approximately 18 months. 

Project level- evaluations might perhaps be conducted for 
$15,000 to $20,000 per project over a period of 12 to 18 months, 
depending on the number of pr\Djects studied. 

The evaluation of alternative interventioi^ strategies as 
described would be a complex undertaking. With, the large number 
of D/R HPEP sites and control sites to be studied, this evaluation 
effort would most likely require $500,000 to $750,000 and 18 to 
24 months; ;o . - ^ 

' - Finally it is estimated 'thq.t the comb^ of a current ' " 

national program evaluation plus full-scale efforts to operation- 

^ ' it 

alize a management information system would cost in the range of 
$500,000 t>o $750, 000. 
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